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ABSTRACT 

This paper considers the case for scientifically based 
research. The paper presents the case in concise outline form, providing a 
rationale; an analogy to medicine; strength of evidenqe argument; and 
thoughts about educational sciences. It asks and answers the questions What 
is EBE? (Evidence Based Education) ; and What is empirical evidence? In 
discussing scientifically based research, the paper considers questions of 
quality; relevance; levels of evidence; randomized trials (asking why 
randomization is critical) ; where the research dollars flow; what the United 
States Department of Education will do; and goals. (NKA) 
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Scientifically Based Research 



Rationale Related documents 

- T ranscript 

• Why scientific research? 

o Research is the only defensible foundation for educational 
practice. 

• If not scientific evidence, then what? 

o Tradition 
o Superstition 
o Anecdote 

Analogy to Medicine 

• Traditions: Bleeding people. 

o Good intentions are not enough. 

• Clinical trials are recent. 

• Why isn't personal experience sufficient? 

• Clinical trials: Only way to really be sure of what works. (Logic) 

• Same rules apply to education: Brain surgery (NAS). 

Strength of Evidence 

• In the meantime, hierarchy of evidence. 

o Not all-or-none 

o Possibly true to probably true versus nothing (NCES 
example) 

® Theory: Evidence-based 

o Knowing why and how something works 
o Key to generalization 
o Pitfalls of theory 
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Educational Sciences 

• No conflict between science and values. 

o Some decisions made on vaiues. 
o Evidence is necessary but not sufficient. 

• Science with a human face. 

• How do we support transiation of research into practice? 

o Suggestions welcome. 



What is EBE? 
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• Best available empirical evidence in making decisions about how to 
deliver instruction 

• Gaps in scientific evidence: Human judgment (bias and wisdom) 

What is empirical evidence? 

• Scientifically based research from fields such as psychology, 
sociology, economics, and neuroscience, and especially from 
research in educational settings 

• Objective measures of performance used to compare, evaluate, and 
monitor progress 

Scientifically Based Research 

• Quality 

o Measures and Methods 
o Scientific merit (double helix) 

• Relevance and Significance 

o Trivial 

o Number affected and severity 

• Two criteria of NSF 

Quality: Levels of evidence 

• All evidence is NOT created equal 

o Randomized trial 

o Quasi-experiment, including before & after 
o Correlational study with statistical controls 
o Correlational study w/o statistical controls (class size, high 
expectations) 
o Case studies 

Randomized Trials: The gold standard 

• Claim about the effects of an educational intervention on outcomes 

• Two or more conditions that differ in levels of exposure to the 
educational intervention 

• Random assignment to conditions 

• Tests for differences in outcomes 

Why is randomization critical? 

• Assures that the participants being compared have the same 
characteristics across the conditions 

• Rules of chance mean that the smart, motivated, experienced, etc. 
have the same probability of being in condition 1 as in condition 2 

• Without randomization, differences between two conditions may 
result from pre-existing difference in the participants, e.g., more 
smart ones in condition 1 

Why is randomization critical? 

Without randomization, simple associations such as between 
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Internet use and science grades have many different interpretations 




Relevance 

• Does the study involve a similar intervention and outcome to those 
of interest? 

• Were the participants and settings representative of those of 
interest? 

• Were enough participants invoived to justify generalization? 
(statisticai inference) 

EBE: How to use existing science 

• Search iiterature (Campbeli Collaboration, Psychinfo, etc.) 

• Screen literature 

o Reievance 
o Quaiity 

• Search for pre-digested evidence 

o Narrative reviews (ERiC digests) 
o Systematic reviews (meta-anaiysis) 

Screening research — Cautions 

• Unconditionai conciusions 

• Conciusions invoiving hypotheticals 

• Conciusions that diverge from evidence 

• Strong caiis to action 

• Mixtures of opinions with evidence 

• Low prestige pubiication outiet 

• Pubiication outiet with ideological agenda 

EBE: How to use objective measures 

• Find or deveiop iocai data 

o measure and measure frequentiy 

• Search state and nationai databases for benchmarks 

o wvw.edtrust.org 
o vww.greatschools.net 
o vww.ses.standardandpoors.com 
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o www.iust4kids.or g 
• Be skeptical 

Evidence-Based Education — Where are we? 

External evidence 




Professional wisdom 
Where the Research Dollars Flow 

• Of 84 program evaluations and studies planned by the Department 
of Education for fiscal year 2000, just one involved a randomized 
field trial 

• 51 were a survey of need 

• 49 had as their purpose program implementation/monitoring, 

• 1 5 were non-randomized impact evaluations. 

• Note: studies could have more than one purpose. 

• Source: Robert Boruch, Dorthy de Moya, and Brooke Synder in 
Robert Boruch and Frederick Mosteller, eds.. Evidence Matters 
(Brookings, 2001). 

What ED will do 

• The What Works Clearinghouse 

o Interventions linked to evidentiary support 
o Systematic reviews 
o Standards for & providers of evaluations 

• Preschool Curriculum Evaluation Research 

• Explanatory Research: Why and How 

• Funding for evaluations of promising innovations in the field 

• Build capacity internally and externally 

Goals 

• ED will provide the tools, information, research, and training to 
support the development of evidence-based education 

• Education across the nation will be continuously improved 

• Wide variation in performance across schools and classrooms will 
be eliminated 

The practice of evidence-based education wiii become routine 

< Previous p age 
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